Project 4 - Term Deposit Sale¶
Problem Statement(Term Deposit Sale)¶
Goal: Using the collected from existing customers, build a model that will help the marketing team identity potential customers who are relatively more likely to subscribe term deposit and thus increase their hit ratio.
Resources Avaliable: The historical data for this project is available in file https://archive.ics.uci.edu/ml/datasets/Bank+Marketing
In [386]:
import pandas as pd
import numpy as np
import warnings
warnings.filterwarnings('ignore')
In [387]:
bank_df = pd.read_csv('bank-full.csv')
In [388]:
bank_df.shape
Out[388]:
Deliverable-1¶
(Exploratory data quality report reflecting the following)
- Univariate analysis a. Univariate analysis - data types and description of the independent attributes which should include(name, meaning, range of values observed, central values (mean and median), standard deviation and quantiles, analysis of the body of distributions/tails, missing values, outliers)
In [389]:
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
In [390]:
bank_df.info()
In [391]:
bank_df.head()
Out[391]:
In [392]:
bank_df.isnull().sum()
Out[392]:
In [393]:
bank_df.describe(include='all').transpose()
Out[393]:
In [ ]:
In [394]:
bank_df['job'].value_counts().plot.bar()
Out[394]:
In [395]:
bank_df['Target'].value_counts().plot.bar()
Out[395]:
In [396]:
bank_df['marital'].value_counts().plot.bar()
Out[396]:
In [397]:
bank_df['education'].value_counts().plot.bar()
Out[397]:
In [398]:
bank_df['default'].value_counts().plot.bar()
Out[398]:
In [399]:
bank_df['housing'].value_counts().plot.bar()
Out[399]:
In [400]:
bank_df['loan'].value_counts().plot.bar()
Out[400]:
In [401]:
bank_df['contact'].value_counts().plot.bar()
Out[401]:
In [402]:
bank_df['month'].value_counts().plot.bar()
Out[402]:
In [403]:
bank_df['poutcome'].value_counts().plot.bar()
Out[403]:
In [404]:
bank_df.describe(include=[np.number]).transpose()
Out[404]:
In [405]:
bank_df.hist(figsize=(25,15))
Out[405]:
Outlier detection analysis
In [406]:
plt.boxplot(bank_df['age'])
Out[406]:
In [407]:
plt.boxplot(bank_df['balance'])
Out[407]:
In [408]:
plt.boxplot(bank_df['campaign'])
Out[408]:
In [409]:
plt.boxplot(bank_df['day'])
Out[409]:
In [410]:
plt.boxplot(bank_df['duration'])
Out[410]:
As part of the EDA Analysis we can see more variations in the balance and age, duaration, days columns which is having outlier data and its has to be cleaned
- Multivariate analysis a. Bi-variate analysis between the predictor variables and between the predictor variables and target column. Comment on your findings in terms of their relationship and degree of relation if any. Presence of leverage points. Visualllize the analysis using boxplots and pair plots, histograms or density curves. Select the most appropriate attributes
In [411]:
sns.pairplot(bank_df)
Out[411]:
As per the pair plot we can see there are multiple numeric columns data is more squead towards 0 and we will be applying normalization or scalling the data.
- Strategies to address the different data challenges such as data pollution, outliers and missing values.
In [412]:
corr = bank_df.corr()
corr
Out[412]:
In [413]:
#handling outlier data
Q1 = bank_df.quantile(0.25)
Q3 = bank_df.quantile(0.75)
IQR = Q3 - Q1
print(IQR)
V = 1.5 * IQR
In [414]:
bank_df.shape
Out[414]:
In [415]:
sns.heatmap(corr,annot=True)
Out[415]:
As pre the correlation plot we can see that pdays and previous , campaign and day features are having more correlation compare to other variables.
In [416]:
#bnk_data = pd.crosstab(index = (bank_df['job']), columns='count')
bnk_data = pd.crosstab(index=bank_df['job'], columns=bank_df['Target'])
bnk_data.plot.bar()
Out[416]:
In [417]:
col_list = ['job','marital','education','default','housing','loan','contact','month','poutcome']
for x in col_list:
bnk_data = pd.crosstab(index=bank_df[x], columns=bank_df['Target'])
bnk_data.plot.bar()
#print(x)
Deliverable - 2¶
(Prepare the data for analytics)
- Load the data into a data-frame. The data-frame should have data and column description
In [418]:
bank_df.info()
In [419]:
#copying data frame
df = bank_df
In [420]:
df.shape
Out[420]:
In [421]:
df.describe(include='all').transpose()
Out[421]:
In [422]:
df.info()
- Ensure the attribute type are correct. If not, take appropriate actions
In [423]:
df.info()# Decision tree in Python can take only numerical / categorical colums. It cannot take string / obeject types.
# The following code loops through each column and checks if the column type is object then converts those columns
# into categorical with each distinct value becoming a category or code.
for feature in df.columns: # Loop through all columns in the dataframe
if df[feature].dtype == 'object': # Only apply for columns with categorical strings
df[feature] = pd.Categorical(df[feature]).codes # Replace strings with an integer
In [424]:
df.info()
- Transform the data i.e. scale / normalize if required
In [425]:
df.loc[df['balance']<0,['balance']] = 0
In [426]:
from sklearn.preprocessing import minmax_scale
df['balance'] = minmax_scale(df['balance'])
df['day'] = minmax_scale(df['day'])
df['duration'] = minmax_scale(df['duration'])
df['campaign'] = minmax_scale(df['campaign'])
df['pdays'] = minmax_scale(df['pdays'])
df['previous'] = minmax_scale(df['previous'])
In [427]:
# plot both together to compare
#fig, ax=plt.subplots(1,2)
#sns.distplot(bank_df['balance'], ax=ax[0])
#ax[0].set_title("Original Data")
#sns.distplot(bank_df['balance_sc'], ax=ax[1])
#ax[1].set_title("Scaled data")
#plt.show()
In [428]:
df[df['balance']<0]
Out[428]:
In [429]:
sns.pairplot(df)
Out[429]:
- create the training set and test set in ratio of 70:30
In [430]:
df.info()
In [431]:
from sklearn.model_selection import train_test_split
In [432]:
X = df.drop('Target', axis=1)
y = df[['Target']]
# split data into train and test/validation sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=101)
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)
In [433]:
y_train['Target'].value_counts()
Out[433]:
In [434]:
y_test['Target'].value_counts()
Out[434]:
Delivrable - 3¶
(create the ensemble model)
build models to predict the variable is_attributed (downloaded). We'll try the several variants of boosting (adaboost, gradient boosting and XGBoost), tune the hyperparameters in each model and choose the one which gives the best performance. In the original Kaggle competition, the metric for model evaluation is area under the ROC curve.
- Write python code using scikitelern, pandas, numpy and other in jupyter notebook to train and test the ensemble model
- First Create a model using standard classification algorithm, Note the model performance
- Use appropriate algorithm's and explain why that algorithm in the comment lines
In [435]:
from sklearn.tree import DecisionTreeClassifier
In [436]:
dt_model = DecisionTreeClassifier(criterion = 'entropy' )
In [437]:
X_train.info()
In [438]:
dt_model.fit(X_train, y_train)
Out[438]:
In [439]:
train_char_label = ['No', 'Yes']
In [440]:
from IPython.display import Image
#import pydotplus as pydot
from sklearn import tree
from os import system
#Bank_Tree_File = open('D:\DW\PGP_AI_ML\Ensemble\Project\bank_tree.dot','w')
#dot_data = tree.export_graphviz(dt_model, out_file=Bank_Tree_File, feature_names = list(X_train), class_names = list(train_char_label))
#Credit_Tree_File.close()
# importance of features in the tree building ( The importance of a feature is computed as the
#(normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance )
print (pd.DataFrame(dt_model.feature_importances_, columns = ["Imp"], index = X_train.columns))
In [441]:
#https://medium.com/@rnbrown/creating-and-visualizing-decision-trees-with-python-f8e8fa394176
#https://stackoverflow.com/questions/36979421/how-can-i-use-conda-to-install-pydotplus
from sklearn.externals.six import StringIO
from IPython.display import Image
from sklearn.tree import export_graphviz
import pydotplus
dot_data = StringIO()
export_graphviz(dt_model, out_file=dot_data,
filled=True, rounded=True,
special_characters=True)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())
Out[441]:
In [442]:
y_predict = dt_model.predict(X_test)
In [443]:
# calculate accuracy measures and confusion matrix
from sklearn import metrics
In [444]:
print(dt_model.score(X_train , y_train))
print(dt_model.score(X_test , y_test))
- Evaluate the model. Use confusion matrix to evaluate class level metrics. i.e Precision and recall. Also reflect the overall score of the model.
In [445]:
print(metrics.confusion_matrix(y_test, y_predict))
In [446]:
#Recall
print(777/(777+820))
#Pression
print(777/(777+857))
#As per this analysis pression and Recall looks so same.
- Advantages and disadvantages of the algorithm
Advantages
- We can group the models together for prediction both in classification and regression model *
Disadvantages
- It will overfit and proven to various errors
- If we build using ensemble method it will become block box very difficult to interpretable
- Build the ensemble models and compare the result with the base model. Note Random forest can be used only with decision Tree
Ensemble Learning - Bagging
In [447]:
from sklearn.ensemble import BaggingClassifier
bgcl = BaggingClassifier(base_estimator=dt_model, n_estimators=50)
#bgcl = BaggingClassifier(n_estimators=50)
bgcl = bgcl.fit(X_train, y_train)
In [448]:
y_predict = bgcl.predict(X_test)
print(bgcl.score(X_test , y_test))
print(metrics.confusion_matrix(y_test, y_predict))
In [449]:
#Recall
print(755/(755+842))
#Pression
print(755/(755+414))
#As per this analysis pression is better than Recall.
Ensemble Learning - AdaBoosting
In [450]:
from sklearn.ensemble import AdaBoostClassifier
abcl = AdaBoostClassifier(base_estimator=dt_model, n_estimators=10)
#abcl = AdaBoostClassifier( n_estimators=50)
abcl = abcl.fit(X_train, y_train)
In [451]:
y_predict = abcl.predict(X_test)
print(abcl.score(X_test , y_test))
print(metrics.confusion_matrix(y_test, y_predict))
In [452]:
#Recall
print(787/(787+810))
#Pression
print(787/(787+852))
#As per this analysis pression and Recall looks almost same.
Ensemble Learning - GradientBoost
In [453]:
from sklearn.ensemble import GradientBoostingClassifier
gbcl = GradientBoostingClassifier(n_estimators = 50)
gbcl = gbcl.fit(X_train, y_train)
In [454]:
y_predict = gbcl.predict(X_test)
print(gbcl.score(X_test , y_test))
print(metrics.confusion_matrix(y_test, y_predict))
In [455]:
#Recall
print(571/(571+1026))
#Pression
print(571/(571+251))
#As per this analysis pression is better than Recall.
Ensemble RandomForest Classifier
In [456]:
from sklearn.ensemble import RandomForestClassifier
rfcl = RandomForestClassifier(n_estimators = 50)
rfcl = rfcl.fit(X_train, y_train)
In [457]:
y_predict = rfcl.predict(X_test)
print(rfcl.score(X_test , y_test))
print(metrics.confusion_matrix(y_test, y_predict))
In [458]:
#Recall
print(690/(690+907))
#Pression
print(690/(690+327))
#As per this analysis pression is better than Recall.
Deliverable-4¶
(Tuning the model)
- Discuss some of the key hyper parameter available for the select alorithm. What values did you initialize these peop.
In [459]:
# parameter grid
param_grid = {"learning_rate": [0.2, 0.6, 0.9],
"subsample": [0.3, 0.6, 0.9]
}
In [460]:
GBC = GradientBoostingClassifier(max_depth=2, n_estimators=200)
In [461]:
from sklearn.model_selection import GridSearchCV
In [462]:
# run grid search
folds = 3
grid_search_GBC = GridSearchCV(GBC,
cv = folds,
param_grid=param_grid,
scoring = 'roc_auc',
return_train_score=True,
verbose = 1)
grid_search_GBC.fit(X_train, y_train)
Out[462]:
In [463]:
cv_results = pd.DataFrame(grid_search_GBC.cv_results_)
cv_results.head()
Out[463]:
In [464]:
# # plotting
plt.figure(figsize=(16,6))
for n, subsample in enumerate(param_grid['subsample']):
# subplot 1/n
plt.subplot(1,len(param_grid['subsample']), n+1)
df = cv_results[cv_results['param_subsample']==subsample]
plt.plot(df["param_learning_rate"], df["mean_test_score"])
plt.plot(df["param_learning_rate"], df["mean_train_score"])
plt.xlabel('learning_rate')
plt.ylabel('AUC')
plt.title("subsample={0}".format(subsample))
plt.ylim([0.60, 1])
plt.legend(['test score', 'train score'], loc='upper left')
plt.xscale('log')
The results show that a subsample size of 0.9 and learning_rate of about 0.2 seems optimal. Also, GradientBoosting and Random Forest has resulted in the highest ROC AUC obtained (across various hyperparameters). Let's build a final model with the chosen hyperparameters.
- Regularization techniques used for the model builing
Regularising the Decision Tree
In [465]:
reg_dt_model = DecisionTreeClassifier(criterion = 'entropy', max_depth = 7)
reg_dt_model.fit(X_train, y_train)
Out[465]:
In [466]:
print (pd.DataFrame(dt_model.feature_importances_, columns = ["Imp"], index = X_train.columns))
In [467]:
from sklearn.externals.six import StringIO
from IPython.display import Image
from sklearn.tree import export_graphviz
import pydotplus
dot_data = StringIO()
export_graphviz(reg_dt_model, out_file=dot_data,
filled=True, rounded=True,
special_characters=True)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())
Out[467]:
In [468]:
y_predict = reg_dt_model.predict(X_test)
In [469]:
reg_dt_model.score(X_test , y_test)
Out[469]:
In [470]:
print(metrics.confusion_matrix(y_test, y_predict))
In [471]:
#Recall
print(564/(564+1033))
#Pression
print(564/(564+300))
#As per this analysis pression is better than Recall.
In [ ]: